Robot utility and interface device

ABSTRACT

Methods and systems are provided for providing real world assistance by a robot utility and interface device (RUID) are provided. A method provides for identifying a position of a user in a physical environment and a surface within the physical environment for projecting an interactive interface. The method also provides for moving to a location within the physical environment based on the position of the user and the surface for projecting the interactive interface. Moreover, the method provides for capturing a plurality of images of the interactive interface while the interactive interface is being interacted with by the use and for determining a selection of an input option made by the user.

FIELD OF THE DISCLOSURE

The present disclosure relates generally to personal robot utility andinterface devices (RUIDs) and more particularly to methods and systemsfor enabling a robot to provide interactive interfaces to users and toprovide accessory devices for interacting with the provided interactiveinterfaces.

BACKGROUND

Virtual assistants such as Apple's Ski, Amazon's Alexa, and Microsoft'sCortana are becoming increasingly popular platforms for users to accessinformation and certain types of content. Virtual assistants allow usersto obtain answers to questions, play audio content, and even purchaseconsumer goods. The ability of current virtual assistants is limited,however, in the type of assistance that may be provided. For example,virtual assistants are generally only able to provide virtual (e.g.,making an electronic purchase) or digital assistance that iscommunicable by audio. As a result, virtual assistants are lacking intheir ability to deliver assistance in the “real world” because they arenot able to perform tasks involving real world objects. Moreover,current virtual assistants are lacking in their ability to interact withusers, typically being responsive to voice commands but limited in otherinput and output modalities.

It is in this context that embodiments arise.

SUMMARY

Embodiments of the present disclosure relate to methods and systems forproviding users with “real world” assistance by a robot utility andinterface device (RUID) and with interactive interfaces forcommunicating with the RUID. In one embodiment, a method is disclosedfor providing an interactive interface by a robot that includes anoperation for identifying a position of a user within a physicalenvironment. The method also identifies a surface within the physicalenvironment for projecting the interactive interface for the user toview. Further, the method provides and operation for moving to alocation within the physical environment based on the position of theuser and the surface for projecting the interactive interface. Moreover,according this embodiment, the method provides operations for capturinga plurality of images of the interactive interface while the interactiveinterface is being interacted with by the user, and for determining,from the plurality of images, a selection of an input option made by theuser. In certain embodiments, the selection of the input option iscommunicated to a server for processing the selection.

In another embodiment, a robot for providing an interactive interface isdescribed. The robot includes a plurality of sensors for obtaining dataassociated with a physical environment. The plurality of sensorsincludes one or more cameras, and the data associated with the physicalenvironment is usable for constructing a map of the physicalenvironment, as well as for identifying a position of a user within thephysical environment and a surface for projecting the interactiveinterface. The robot additionally includes a processor configured forprocessing the data associated with the physical environment fordetermining a location within the physical environment where the robotis to move for said projecting the interactive interface based on themap of the physical environment, the position of the user, and alocation of the surface. According the present embodiment, the robotalso includes a locomotion unit for moving the robot to the locationwithin the physical environment, a graphics unit for rendering theinteractive interface, and a projector system configured for projectingthe interactive interface onto the surface. Moreover, the one or morecameras are configured to capture a plurality of images of theinteractive interface while the interactive interface is beinginteracted with by the user, and the processor is further configured toprocess the plurality of images for determining a selection by the userof an input option.

In another embodiment, a method for providing a gaming session by arobot is described. The method includes operations for receiving aninstruction from a user to execute a video game and for identifying aposition of a user within a physical environment. The method alsoincludes operations for identifying attributes of a surface within thephysical environment for projecting the video game. According to thepresent embodiment, the method provides operations for determining agame controller from a plurality of game controllers to be used by theuser for interacting with the video game and for establishing a wirelessconnection with the game controller. Further operations have the robotmoving to a first location proximal to the user for delivering the gamecontroller to the user and then to second location for projecting agraphical output of the video game based on the position of the user andthe attributes of the surface. Moreover, the method has an operation forexecuting the video game and projecting the graphical output of thevideo game onto the surface, wherein the game controller enables theuser to interact with the video game.

In some embodiments, determining the game controller from the pluralityof game controllers is based on a request made by the user, a preferencesetting of the user, a prediction, a degree of randomness, or aspecification provided by the video game.

In further embodiments, the game controller may be a detachable partthat is mounted to the robot. For example, the game controller may be arobotic arm of the robot that is configured to allow the user tointeract with the video game by detaching the robotic arm or portionsthereof.

In various embodiments, the robot is configured to be ridden by an HMDuser interacting with a virtual reality scene. In these embodiments, therobot cooperates with the HMD to provide movements in the real worldspace synchronous to movements in the virtual space of the VR scene.

Other aspects of the disclosure will become apparent from the followingdetailed description, taken in conjunction with the accompanyingdrawings, illustrating by way of example the principles of thedisclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure may best be understood by reference to the followingdescription taken in conjunction with the accompanying drawings inwhich:

FIGS. 1A and 1B show overall flows of methods for providing interactiveinterfaces projected onto surfaces, according to certain embodiments.

FIG. 2 shows a robot presenting an interactive interface to a user formaking a selection by projecting the interactive interface onto a wall,according to one embodiment.

FIGS. 3A-3F show various embodiments of robot utility and interfacedevice (RUID) enabled interactive interfacing.

FIGS. 4A-4C show various embodiments of movement associated with a RUID.

FIG. 5 shows an overall flow of a method for providing a user with avideo game and a game controller for interacting with the video game,according to one embodiment.

FIGS. 6A-6C show a user being provided with a video game and a gamecontroller for interacting with the video game in response to a command,according to one embodiment.

FIG. 7A shows a user being provided with a game controller in responseto a command for interacting with a video game being presented on adisplay, according to one embodiment.

FIG. 7B shows a user being provided with a game controller in responseto a command for interacting with a virtual reality environmentpresented via head mounted display (HMD), according to one embodiment.

FIGS. 7C and 7D illustrate RUID-enabled assistance for a user playing avideo game, according to various embodiments.

FIG. 8 shows an overall flow of a method for providing a user with oneof a plurality of variably structurally configured game controllers forinteracting with a video game.

FIGS. 9A-9C illustrate a RUID-enabled gaming session wherein adetachable arm of the RUID is configured according to a selection madeby a user and used for interacting with a video game.

FIG. 10A-10D show various embodiments of RUIDs and RUID enabledactivity.

FIG. 11 shows various configurations of an arm of a RUID that may beused as game controllers.

FIGS. 12A-12E illustrate conceptual models for orienting a videoprojector of a robot to have a projection axis that is perpendicular toa surface for projection.

FIG. 13 shows an additional embodiment of a RUID and some associatedcomponents.

DETAILED DESCRIPTION

The following embodiments describe methods, computer programs, andapparatuses for enabling a robot to provide interactive interfaces forpresentation of content and to supply interaction devices to forinteracting with the content. It will be obvious, however, to oneskilled in the art, that the present disclosure may be practiced withoutsome or all of these specific details. In other instances, well knownprocess operations have not been described in detail in order to notunnecessarily obscure the present disclosure.

Virtual assistants such as Apple's Ski, Amazon's Alexa, and Microsoft'sCortana are becoming increasingly popular platforms for users to accessinformation and certain types of content. Virtual assistants allow usersto obtain answers to questions, play audio content, and even purchaseconsumer goods.

The ability of current virtual assistants is limited, however, in thetype of assistance that may be provided. For example, virtual assistantsgenerally only excel at providing informational, digital, and electronicassistance, but fall short in other types of assistance that a user mayneed. For example, current virtual assistants are not able to retrieve aphysical good to a user, modify the physical good for the user, orrelocate itself according to a location of the user. That is, currentvirtual assistants are limited in their ability to perform “real world”tasks such as those involving spatial navigation and physicalmanipulation. As a result, virtual assistants are lacking in theirability to deliver assistance in the “real world” because they are notable to perform tasks involving real world objects.

Virtual assistants are further lacking in their ability to provide userinterfaces for users to interact with, typically allowing only voiceinput and output. There are various types of information and contentthat are less effectively transmitted by a virtual assistant's voice.Other types of content simply cannot be transmitted via voice.

Moreover, the functionality of virtual assistants is limited in thecontext of media presentation environments such as for playing a videogame. For example, virtual assistants are generally not able to assist aplayer in setting up a video game session or assist a player duringinteraction with the video game. Often, limitations to virtualassistants in the abovementioned contexts stem from a lack of a realworld presence, a lack of an awareness of real world contents, a lack ofmobility, a lack of an ability to manipulate physical objects, a lack ofconnectivity to a media player or gaming device, and a lack of a realworld interface.

Systems, methods, and apparatuses described here enable real worldassistance to users by completing real world tasks and by providing realworld user interfaces for users to interact with. According to oneembodiment, a robot utility and interface device (RUID) is provided thatcan provide a real world interface at various locations for a user tointeract with. For example, the RUID may identify a position of a userwithin a physical environment as well as a nearby wall for projecting auser interface. The RUID may then move about the physical environment toa location where it may project the user interface onto a region of thewall that is convenient for the user to look at. In some embodiments,the user is enabled to provide input to the interface by performingcertain types of gestures. For example, the RUID may be configured todetect that a user has touched the wall at a location corresponding to areactive element such as a button. In these and other embodiments, theRUID is additionally configured to detect other gestures made by theuser, including a scrolling gesture, a zooming in or out gesture, a“back” gesture for going back to a prior page or screen, a “switch”gesture for switching between applications and/or interfaces, as well asother gestures.

According to various embodiments, a RUID captures a plurality of imagesof the user interface while the user is interacting with the userinterface. For example, the RUID may obtain images where the user istouching or pointing to a region of the wall on which user interface isbeing projected on. The RUID is then able to use image analysis todetermine what part of the projected user interface is being “touched”or pointed at. In response to the user's input, the RUID may thenprovide additional content via projection. As a result, methods andsystem described herein provide a technological benefit over currentvirtual or machine assistants by enabling the user to interface with auser interface that automatically adapts to a user's location in a realworld space by projecting the user interface at a location that isconvenient for the user to interact with.

According to various embodiments, the RUID is capable of providing realworld assistance to a user, either automatically or in response toinstruction. In one embodiment, the RUID is able to interpretinstructions of a user to initiate a video game. The RUID identifies aposition of the user within a physical environment as well as a suitablesurface such as a wall for projecting the video game. In someembodiments, the RUID will identify a game controller from a pluralityof game controllers that is to be used by the user. The RUID may do sobased on one or more of an instruction of the user, a specification of agame controller contained by the video game, a preference setting of theuser, a history of game controller use, a degree of randomness, and gamecontroller use data for other users obtained via the Internet. Based onan identification of the game controller to be used, the RUID may thenmove to a vicinity of the user to deliver the game controller to theuser. In some embodiments, the RUID is capable of delivering objectsusing a robot arm having a grasping mechanism. Further, in someembodiments, the plurality of game controllers or a portion thereof maybe supported or docked onto the RUID. The RUID may then grasp one gamecontroller of the plurality of game controllers and deliver it at aconvenient location and height for the user. Moreover, the RUID isenabled to move to a different location for projecting the video game,if necessary or desired.

In various embodiments, the RUID is enabled to assist a user during theuser's interaction with a video game. For example, if the user isplaying a video game on a console system that is being displayed on adisplay, RUID is able to provide visual assistance by projecting onto awall near the display a supplemental display. In the supplementaldisplay, the RUID may provide various types of visual assistance to theuser, including instructions, tips or tricks, hacks, maps, directions,walk-through guide content, or the like.

In certain other embodiments, the RUID may provide real world assistancein addition to visual assistance. For example, if a user requests realworld assistance in defeating an enemy character, the RUID is able towield and control a controller (e.g., with its robotic arm and grasper)such as an aiming device to assist in defeating the enemy character.That is, the RUID may act like a second player by helping the user inthe game via a game controller. As a result, a RUID embodies atechnological benefit from its ability to provide real world assistanceto users that is not seen in current virtual assistants. Thus, atechnological improvement to current virtual assistants is provided bythe systems and methods described herein.

As used herein, “audio assistance” or “verbal assistance” is taken tomean assistance that is delivered to a person via a mode of speech orother audio. For example, certain embodiments of RUIDs are able toprovide audio assistance by providing a verbal answer to a questionposed by the user. In another embodiment, a RUID may provide audioassistance by presenting a surround sound component associated withmedia content to a user viewing said media content on an entertainmentsystem.

As used herein, “visual assistance” is taken to mean assistance that isdelivered to a person via a mode of display. For example, certainembodiments of RUIDs are capable of performing visual assistance byprojecting content for display on a surface for a user to perceive.

As used herein, “real world assistance” is taken to mean assistance thatis delivered to a person via a mode of physical and/or spatial action,application of force, transportation, manipulation, and/or movement. Forexample, certain embodiments of RUIDs are capable of providing realworld assistance by delivering a game controller to a user.

It should be noted that RUIDs described here may be capable of providingeach of verbal assistance, visual assistance, and real world assistancein combination or simultaneously for various situations.

FIG. 1A shows an overall flow of a method for providing an interactiveuser interface that is projected onto a surface such as a wall.According to the embodiment shown, the method in operation 100identifies a position of a user within the physical environment. As willbe discussed in further detail below, identification of a user may beaccomplished using a plurality of sensors and cameras.

In operation 102, the method identifies a surface within the physicalenvironment for projecting a user interface based on the position of theuser. The surface may be one associated with a wall, a door, a ceiling,a piece of furniture, a post, an object held by the user, or otherobject. In operation 102, or in a separate operation, the method mayanalyze the surface for color, texture, shape, contours, geometry, size,reflectivity, refractivity, gain, position and orientation in a physicalenvironment relative to the user and/or other optical and geometricproperties. In some embodiments, a system for implementing the methodshown in FIG. 1A may perform a number of tests on various surfaces byprojecting a test sequence on said various surfaces and capturing theresulting projection. The system may construct a map of the varioussurfaces within the physical environment, the map including the locationas well as the optical and geometric information of the surfaces. Thesystem may then identify a surface that is suitable for projecting theuser interface based on the map and the position of the user.

The method shown in FIG. 1A then flows to operation 104, which providesfor moving within the physical environment to a location based on theidentified surface and the location of the user. For example, afteridentifying a suitable surface for projecting the user interface, asystem implementing the method shown in the FIG. 1A may not be in anoptimal position to project the user interface onto the surface. Thesystem may be too close or too far from the surface, or there may beobjects occluding an optical path of the projection. Moreover, a videoprojector of the system may not be “square” with the surface. That is, aline or central axis of projection describing a direction of light raysemitted by the video projector may not be orthogonal to a planedescribing the surface. As a result, the system may move to a locationin the physical environment and orient the video projector such that thecentral axis of projection is more orthogonal to the surface.

In certain other embodiments, the system may compensate for a centralaxis of projection that is less than orthogonal by modifying images tobe projected. For example, if there is a skew or obliqueness associatedwith a central axis of projection that results or would result in adistortion of the projected image, certain embodiments may correct forsuch distortion by modifying images in a manner that is opposite andproportional to the degree and directionality of distortion. In someembodiments, a combination of relocation and distortion correction maybe implemented together by a system.

The method shown in FIG. 1A then flows to operation 106, which serves toproject the user interface onto the surface. As a result, the userinterface is made visible to the user, and the user may proceed tointeract with it. There are a number of ways for a user to interact withthe projected user interface, some including a touching of the surfacethat the user interface is being projected onto for making an input.According to other embodiments, it is contemplated that other handgestures may generate input to the user interface, the hand gesturesincluding a pointing of a finger, a scrolling gesture, a sign made byhand, or other hand and finger positioning and movements.

The method of FIG. 1A then flows to operation 108, which serves tocapture a plurality of images of the user while interacting with theinterface during the projecting the user interface on the surface. As aresult, the method is able to obtain information on how the userinteracts with the user interface. Moreover, in operation 110, themethod determines from the plurality of images that an option isassociated with a selection by the user. For example, if a user makes agesture for selecting a particular option that is presented in the userinterface, operation 110 is able to determine that the user made such aselection. In the embodiment shown, an indication may be provided to theuser interface indicating that the option has been selected, accordingto operation 112. As a result, the user is given feedback that theselection of the option has been registered. In some embodiments, themethod is able to provide additional content associated with the optionin response to detecting that the user has selected the option. Theseembodiments will be discussed in further detail below.

FIG. 1B shows a flow of methods for providing interactive interfacesprojected onto surfaces. In 120, an instruction is received from a userfor executing an application. In 122, a position and point of gaze (POG)of the user is identified within a physical environment. The method in124 includes identifying a surface within the physical environment forprojecting a user interface associated with the application based theposition and POG of the user, the user interface including a pluralityof options. In 126, the method includes projecting the user interfaceonto the surface. In 128, the method includes capturing a plurality ofimages of the user interacting with the user interface during theprojecting the user interface on the surface. In 130, the methodincludes determining, from the plurality of images, an option associatedwith a selection of the user. In 132, the method includes providing, tothe user interface, additional content associated with the optionselected.

FIG. 2 shows a robot 202, such as a RUID, presenting an interactiveinterface 204 to a user 200 for making a selection by projecting theinteractive interface 204 onto a wall 201. The interactive interface 204is shown to include instruction to select between options A, B, C, andD. In response, the user 200 is shown to make a contact 206 with thewall 201 in a region associated with an Option A button 208 of theinteractive interface 204. During the interaction of user 200 with theinteractive interface 204, the robot 202 is shown to capture a pluralityof images 214 of the interactive interface 204 with an image capturedevice 212 having a field of view 224 that is directed toward theinteractive interface 204.

The plurality of images 214 is shown to capture the user 200 placing anindex finger on the Option A button 208 of the interactive interface204. Further, image analysis logic 216 is shown to have determined thatoption A 218 has been selected. As a result, robot 202 is able todetermine the selection made by user 200 with respect to the interactiveinterface 204. Moreover, the user 200 may be given an indication 226that his input has been received.

Robot 202 is also shown to have a sound capture device 220 for capturingsounds produced in the physical environment. In some embodiments, thesound capture device 220 is able to capture the sound of user 200 makingcontact 206 with wall 201 and use sound analysis 222 to determine thatcontact 206 has been made.

FIGS. 3A-3F illustrate various embodiments of a robot utility andinterface device (RUID) providing visual assistance to a user inresponse to a command. As mentioned above, there are many types ofassistance a user may request that are not conducive to verbal response.The embodiments shown in FIGS. 3A-3F demonstrate a few exemplary typesof assistance that a RUID is capable of providing in a visual format.

FIG. 3A shows a user requesting directions. In response, the RUIDprojects a map including a suggested route to take onto a wall thatproximal to the user. In FIG. 3B, the RUID projects a graphicalrepresentation of the forecast in response to a user's request. FIG. 3Cshows the RUID projecting a map of nearby restaurants in response to auser's request. In FIG. 3D, the RUID displays highlights of a baseballgame in response to a user's request. FIG. 3E shows a user requestingthat the robot display products that are compatible with a device thatis already owned. The robot is shown to obtain information about thedevice such as a make and model by capturing images of the device. Therobot then proceeds to display a plurality of items in response to theuser's request. FIG. 3F shows a user requesting that the robot show howto fix a bicycle wheel. The robot, after having identified a type, make,and/or model of the wheel, may then display a video showing the user howto fix the wheel. As a result, the robot is able to provide visualassistance for many types of circumstances and in response to many typesof requests. The embodiments shown in FIGS. 3A-3F are meant to beexemplary, however, and not limiting to the types of interfaces andcontent that a robot according to the systems and methods presented hereare capable of. Additionally, it is contemplated that a robot accordingto the systems and methods provided herein are enabled to capture userinput during interaction with the projected displays shown in FIGS.3A-3F (please see FIGS. 1 and 2).

FIGS. 4A-4C demonstrate various ways in which a RUID may performmovements in response to a user's movement. For example, in FIG. 4A, auser is shown to have moved longitudinally along a wall for somedistance. In response, the RUID is shown to have moved a similardistance longitudinally while continuing to project for the display ofcontent. The RUID may detect movement of the user in a number of ways,for example via image capture, depth sensors, proximity sensors,infrared (IR) sensors, stereoscopic cameras, and the like. As a result,the user is given continuous view of the content being displayed viaprojection while they are moving within a physical space without havingto physically transport or even provide instruction to a display device.

In FIG. 4B, a user is shown to have traveled some latitudinal distanceaway from the wall on which content is being displayed via projection.The robot is shown to have made the size of the display some size largerthat is, for example, proportional to the distance traveled by the user.In some embodiments, the robot may be able to change a size of thedisplay area by using a screen size adjust feature associated with aprojector for projecting the display. In other embodiments, the robotmay increase a distance from the wall on which the display is beingprojected to increase the screen size. In the embodiment of FIG. 4B, acombination of screen size adjustment via the projector and a movementaway from the wall is shown to occur. As a result, the user is given aview of the display after having moved similar to that of before havingmoved. That is, the display is made to look just as big after the userhas moved away from the wall. Moreover, the robot may adjust the displaysize in synchrony with the movement of the user. Thus, robotsimplemented according to system and embodiments described here enable adisplay that is further able to adjust an apparent size (e.g., how bigor small the display appears) of the display in response to a user'sdistance away from a display. The ability to adjust an apparent size ofa display in synchrony and in accordance with a user's distance awayfrom the display provides one of many technological benefits of systemsand methods presented here, and represent a technological improvement tocurrent display technologies (e.g., flat panel displays, videoprojectors, handheld displays etc.).

FIG. 4C shows a user who moves from a first position that is proximal toa first wall to a second position that is proximal to a second wall. Inresponse to the user's movement, the robot is shown to also have movedto a second position for projecting onto the second wall. During therobot's movement, a projector of the robot is also oriented such that itis directed toward the second wall. As a result, in addition to beingable to move a display of content within a given surface in accordancewith movement of a user, a robot in accordance with the systems andmethods presented here are further able to move the display of contentto different surfaces depending on where the user is moving.

FIG. 5 shows an overall flow of a method for executing and presenting avideo game to a user as well as providing the user with a gamecontroller for interacting with the video game, according to oneembodiment. In operation 500, the method provides for receiving aninstruction from a user to execute a video game. In response, the methodprovides for operation 510, which serves to identify a position of theuser within a physical environment such as a living room orentertainment room, and for operation 520, which serves to identify asurface within the physical environment for projecting a graphicaloutput of the video game based on the position of the user. For example,operation 510 may determine that a user is sitting on a couch at adistance of 10 feet away from a wall. Operation 520 would serve toidentify the wall as a suitable surface to project the video game onto.

Further the method provides for an operation 530 that identifies a gamecontroller to be used by the user for interacting with the video game.There are many types of game controllers used by video game players forinteracting with video games. Some of them include traditional handheldcontrollers like the Dual Shock controller, aiming controllers,wand-like controllers, joystick controllers, sword controllers, steeringwheel controllers, and many more. According to the method shown in FIG.5, operation 530 is able to determine a game controller from a pluralityof game controllers that is the user is to use for interacting with thevideo game. For example, operation 530 may identify which of the gamecontrollers to used based one or more of the following: an explicitinstruction received from the user, a specification associated with thevideo game, a preference setting of the user, a history of gamecontroller use by the user, game controller user data for gamecontroller usage tendencies of other users, machine learning directed atlikelihood that a user will want to use a certain game controller, aswell as other factors. For example, operation 530 may determine that aDual Shock game controller is to be used based on one or more of theaforementioned factors. Moreover, although now shown, operation 530 mayalso serve to activate the identified game controller (e.g., power iton) and establish a wireless connection between the identified gamecontroller and, for example, a robot implementing the method of FIG. 5(e.g., via Bluetooth, Wi-Fi, near field communication, or othercommunication standard).

The method of FIG. 5 also includes operation 540 providing for moving toone or more locations within the physical environment for providing thegame controller to the user. In some embodiments, a robot such as a RUIDimplementing the method shown in FIG. 5 may move within reachingdistance of the user so that the user may pick the identified gamecontroller, which may be already powered on and connected with therobot. In other embodiments, a robot carrying out the present method mayinclude a robotic arm and grasper for picking up the identified gamecontroller and “handing” it to the user. The plurality of gamecontrollers may be in different regions of the physical space, or may beat a central location such as a docking station for charging. As aresult, they may need to be retrieved. In certain other embodimentscontemplated, the plurality of game controllers or a portion thereof maybe stationed or docked onto the robot, or may be modular parts of therobot's structure.

Also shown in FIG. 5 are operations 550 and 560, which serve,respectively, to move to one or more locations for the projecting of thevideo game onto the identified surface, and to execute and project thegraphical output of the video game onto the surface for enabling theuser to view and interact with the video game via the game controller.For example, after handing the identified controller to the user inoperation 540, the robot implementing the method shown may need to moveto a different location better suited for projecting the video game fordisplay. As a result of the method, a user may be provided with all ofthe components for playing a video game in a standalone device forimplementing the present method without, for example, needing to performany action typically related to setting up a gaming session.

FIGS. 6A-6C illustrate a RUID 601 enabled gaming session having a user600 provided with a game controller 610 for interacting with a gameexecuted on the RUID 601. In FIG. 6A, the user 600 is shown to requestthat Final Fantasy VIII video game be initiated. RUID 601 receives therequest and determines a gaming controller to be used for the videogame. A number of game controllers are docked on RUID 601, including agame controller 610, a sword controller 612, and an aim controller 614.As shown in FIG. 6B, the RUID determines that game controller 610 is thegame controller to be used by user 600.

Additionally, in the embodiment shown, the RUID 601 includes a roboticarm 606 and grasper 608 that is operable to grasp the game controller610 from a position on the RUID 601 where it is docked and translocateto within a distance that is convenient for the user 600 to grab thegame controller 610.

In FIG. 6C, RUID 601 is shown to be executing and projecting thegraphical output of the video game 616 onto wall 602. In somecircumstances, RUID 601 will move from a location for delivering thegame controller 610 to a different location for projecting the videogame. In other embodiments, the RUID 601 may move to a differentlocation for projecting the video game. In either case, the user 600 isprovided with components for interacting with a video game withouthaving to perform much of the typical tasks involved with setting up agaming session.

FIG. 7A-7D illustrate different ways in which a RUID may provide realworld assistance a context of playing video games. In FIG. 7A a user isshown to be playing a video game displayed on a display. The video gamemay be executed by the RUID shown, or by a game console, personalcomputer, or a remote gaming server. The user is shown to request theaim controller, and the RUID is shown to hand the aim controller to theuser in response. As a result, real world assistance is provided to theuser, such as for wanting to switch game controllers. The user benefitsby not having to pause the video game and physically retrieve the gamecontroller, or wait for a connection between the retrieved gamecontroller and the gaming device hosting the video game.

FIG. 7B illustrates a user playing a virtual reality video gamepresented via a head mounted display (HMD). The user is provided with anaim controller in response to his request. In some embodiments, the usermay be made aware of the location of the aim controller via the HMDdisplay by having the cameras of the HMD capture images of the RUID andthe aim controller. The HMD may display visual representation in realworld coordinates of the aim controller to the user so that the user mayreach out and grab the aim controller without having to remove the HMD.

FIG. 7C shows a user who is need of help or assistance in defeating anenemy. The user is shown to request that the RUID shows the user how todefeat the monster. In response, the RUID provides visual assistance byprojecting a display that includes an image or clip on how to defeat themonster. Additionally, the RUID may provide verbal assistance, includingverbal instructions for how to defeat the monster.

In FIG. 7D, a user is shown to request that the RUID help the userdefeat the monster. Thus, instead of showing the user how to defeat themonster on a projected display (e.g., as in FIG. 7C), the RUID is shownto grasp an aim controller and use it to help the user defeat themonster. Thus, it is contemplated here that a RUID may provideassistance to the user in a way that mimics a way a second user would.Namely, it is contemplated that RUIDs are not only able to retrieve andtransport game controllers but also to use them in a way a person might.As a result, methods and system described here enable robots such asRUIDs to assist users in the context of video gaming by providingassistance that requires “real world” manipulation (e.g., application offorce) of real world objects (e.g., the game controller), in addition toproviding verbal and visual assistance. The user is benefited in hisgaming experience in ways that current methods of assisting users (e.g.,virtual assistants) do not appear to be able to provide.

According to some embodiments, the RUID may also provide real worldeffects that correspond with virtual effects that occur within VR scene.For example, if there is a gust of wind that blows in a particulardirection a game, the RUID may be enabled to produce a wind effect byblowing wind in the particular direction to the user in the real world.Thus, it is contemplated that the RUID is able to augment virtualeffects provided by the HMD with real world effects.

In certain other embodiments, the RUID is configured to be sat on orridden by an HMD user being presented with a VR scene. The RUID isenabled to interact with the HMD and/or computing device that executesthe VR scene such that it moves the HMD user atop the RUID in a mannerthat corresponds with the HMD user's movement in the VR scene. Forexample, if a VR scene includes a segment that has the user moving tothe right for 3 feet in the virtual space, the RUID may be synchronouslymove itself and sitting HMD user 3 feet to the right in the real world.Thus, the RUID is able to provide additional real world effects thatenhance VR effects. As a result, it is contemplated that the RUID isable to provide a more immersive virtual experience for an HMD user.

FIG. 8 shows and overall flow of a method for providing a user with oneof a plurality of variably configured robotic arms to be used as a gamecontroller. FIG. 8A includes an operation 800 for executing a video gameincluding a user interface. Operation 810 serves to project the userinterface onto a surface for display to the user. The user interfaceenables the user to make a selection of an object to be used forinteracting with the video game. Often, video games will enable users tohave a tool or weapon or other piece of equipment for interacting withthe video game. The user interface of operation 810 enables the user tochoose a piece of equipment that the user wants to use for interactingwith the video game. Operation 820 serves to identify the selection madeby the user.

The method of FIG. 8 then flows to operation 830, which provides forconfiguring a detachable arm of a robot such as a RUID according to theselection made by the user, where the detachable arm is able to be heldby the user. According to some embodiments the detachable arm of therobot may be configured into a plurality of shapes or modules to be usedto interact with the video game. The robot may configure the arm into acertain shape, and in some embodiments, engage mechanisms for locking ormaking rigid the detachable arm in the desired shape. The user is thenable to detach the detachable arm from the robot for use in the game.

According to various embodiments, the detachable arm is formed byoperation 830 into a shape that corresponds to a shape of the selectedpiece of equipment identified in operation 820. For example, if theselection identified in operation 820 is for a sword, the operation 830may configure the detachable arm to have a straight elongated shape. If,on the other hand, the selection is for a shield, operation 830 mayconfigure the detachable arm to be in a different shape. As a result,the user is provided with a plurality of game controller configurationsthat accord in shape with the piece of equipment the user chooses touse. Further, operation 840 serves to execute and project the video gameonto the surface for enabling the user to interact with the video gameusing the detachable arm in the configured shape.

According to certain other embodiments, a robot may configure adetachable arm for a user to use in a game that is executing on acomputing device that is of the robot and not displayed by the robot.For example, in these embodiments a robot may connect to a computingdevice such as a game console, a personal computer, a tablet, or aremote game server for receiving information about a video game beingplayed by a user. It may also receive information relating to a piece ofequipment the user selects to use for interacting with the video game.The robot may then configure a detachable arm according to the receivedinformation for the selected piece of equipment. As a result, the robotmay provide various game controller configurations for users interactingwith other video game platforms.

FIGS. 9A-9C illustrate a RUID-enabled gaming session wherein adetachable arm of the RUID is configured according to a selection madeby a user and used for interacting with a video game. In FIG. 9A, a RUIDis shown to display a user interface via a projector for a user toselect between different pieces of equipment. In response, the user isshown to select a golf club by indicating his selection with his finger.

In FIG. 9B, a detachable arm of the RUID is shown to have beenconfigured into a different shape. In particular, the detachable arm isshown to have been elongated into having a substantially straight shaft.Moreover, a longitudinal axis of a grasper of the detachable arm isshown to be configured to be at an obtuse angle relative to thelongitudinal axis of the elongated shaft portion of the detachable arm.As a result, the detachable arm is configured into a shape that accordswith a shape of the selected equipment. Namely, the detachable arm maybe used as a golf club game controller by the user. After the detachablearm is configured according to the user's selection, the user interfaceindicates that the equipment is ready for the user to use.

In FIG. 9C, the detachable arm of the RUID having been configured tofunction as a golf club is shown to be used by the user to play in agolf game. In various embodiments, the user input made through thedetachable arm (e.g., swinging the ‘golf club controller’) may beobtained via a plurality of sensors located within the physicalenvironment and/or on the RUID for detecting location and movement ofthe detachable arm. For example, the detachable arm may include aplurality of light emitting diodes (LEDs) on its surface that aretracked by cameras in the physical environment. Moreover, the detachablearm may include inertial sensors to sense movement and acceleration ofthe detachable arm through space. As a result, the RUID executing thevideo game is enabled to transduce and convert physical movement of thedetachable arm, as well as other game controllers, into virtualmovement. For example, the RUID is able to determine a virtual responseto the user's swinging of the ‘golf club controller’ from analyzing thegolf club game controller's position in space at different times.Depending on the mechanics of the user's swing of the golf clubcontroller and the mechanics of the video game, the RUID will present acorresponding virtual response displayed on the projected display.

It should be noted the detachable arm may have been configureddifferently had the user selected a different piece of equipment to usefor the video game. For example, had the user selected a tennis racket,the configuration of the detachable arm may have been made to resemblean overall feel of a tennis racket. Thus, the detachable arm may havebeen disjointed such that only a portion of the detachable is used asthe game controller (see FIG. 11).

FIG. 10A illustrates an embodiment of a robot utility and interfacedevice (RUID) 1000 for enabling methods presented here. The RUID 1000 isshown to include wheels 1002 a-1002 c for moving within a physicalenvironment. The RUID 1000 is also shown to dock an aim controller 1004,a Dual Shock controller 1006, and a sword controller 1008. In someembodiments, the RUID 1000 is able to charge one or more of controllers1004-1008 with an on-board battery (not shown).

RUID 1000 is also shown to include a video projector 1010 for projectinguser interfaces and media content such as video games onto surfaces fordisplay to a user. A sensor module 1012 enables the RUID 1012 to map andnavigate the physical environment, identify a user's position, andidentify surfaces for displaying content. The sensor module 1012 mayinclude a number of sensors, including proximity sensors, depth sensors,cameras, stereoscopic cameras, infrared sensors, magnetic sensors,radar, among others. The RUID 100 is also shown to have a robotic arm1014 and grasper 1016 for providing real world assistance such asretrieving and delivering game controllers to a user, helping a userexchange game controllers, ‘putting away’ game controllers. Moreover, asdiscussed above, the robotic arm 1014 may be configurable into differentshapes and detachable for use as a game controller for interacting witha video game.

Although not shown, in some embodiments the RUID may be used as afootrest by a user. In certain embodiments, a footrest attachment orcushion may be compatible with the RUID and the arm and grasper of thecushion. As a result, the RUID may support a foot rest cushion at aheight that is adjustable and comfortable to a user who is sitting in achair or on a couch.

According to the embodiment shown, RUID 100 also includes a microphone1018 for capturing voice commands of a user and other sounds of thephysical environment.

FIG. 10B illustrates an additional embodiment of a RUID 1000 having agrasper hand 1020 that is also contemplated here. The grasper hand 1020is shown to take on a shape that is similar to a human hand, and as aresult, may interact with the user to mimic behavior of a human hand. InFIG. 10C, an additional embodiment of a RUID 1000 is shown to have atelescoping device 1022 for elevating certain portions of the RUID 1000.For example, the telescoping device 1022 may be able to elevate aprojector, a sensor module, and/or a camera, as well as othercomponents. Moreover, the telescoping device 1022 may also be able toelevate a robotic arm having a grasper.

In FIG. 10D, a user 1024 is shown to be wearing a head mounted display(HMD) 1026 for presentation of a virtual reality scene 1030. In thevirtual reality (VR) scene 1030, user 1024 is shown to attempt to shakehands with a character 1032. In some embodiments, the VR scene 1030 maybe executing on the RUID 1000, which provides images associated with theVR scene 1030 to the HMD 1026 via wireless connection. Concurrently,while the user reaches his hand out to perform a virtual handshake1028′, the RUID 1000 provides a real world hand shake 1028 with the user1024. As a result, the user 1024 is given a physical sensation of a realworld handshake 1028 that corresponds to his virtual handshake 1028′.

According to certain embodiments, a robot is able to track the positionof the hand of the user 1024, as well as the location of the robot'sgrasper hand 1020. Additionally, the robot has information related towhere the hand of the VR character 1032 should be. As a result, therobot is able to place its grasper hand 1020 at a location relative tothe user 1024 that corresponds to where the user 1024 perceives the handof the character 1032 to be in a virtual space. As a result, when theuser 1024 reaches out his hand, the grasper hand 1020 is in position tomeet it for a real world handshake 1028.

Although a left-handed grasper hand is shown in FIGS. 10B-D, it is to beunderstood that right-handed grasper hand may also be used in differentembodiments. Moreover, some embodiments may have RUIDs with two or morerobotic arms, each of the arms having a grasper hand.

FIG. 11 illustrate various configurations of an arm of a RUID that maybe used as game controllers. For example, configuration 1102 is providedby detaching the grasper of the arm. A game controller of configuration1102 may be used as a ping-pong paddle, a wand, a pointing or aimingdevice, or other one-handed device. A game controller of configuration1104 may be used as a baseball bat, a cricket bat, a sword, a lightsaber, a pole, a staff, or other one or two-handed device. A gamecontroller of configuration 1106, as shown in FIG. 9C, may be used as agolf club. Additionally, however, the configuration 1106 may also beused as shooting device, a cannon, a mortar, a rocket propelled grenade(RPG) launcher, a scythe, or other device. A game controller ofconfiguration 1108 may be used as a tennis racket, a racquet ballracket, a baton, a dagger, a shield, a wand, or other device. A gamecontroller of configuration 1110 may be used as an aiming device, orother device. A game controller of configuration 1112 is shown to be ofa trapezoidal shape, having the grasper grasp a portion of the shaft ofthe arm. A game controller of such a configuration may be used as ashield, an aiming device, a steering wheel, a crossbow or other bow, ashooting device, an aiming device, or other type of equipment. Ofcourse, there are many additional configurations of the robot arm thatare possible, but not illustrated in FIG. 11. As a result, theconfigurations shown in FIG. 11 are meant to be exemplary and limiting,and the many additional configurations do not depart from the scope andspirit of the embodiments described here.

FIG. 12A-12E illustrate conceptual models for orienting a videoprojector 1201 of a robot 1200 to be square with a surface to beprojected on. For example, FIG. 12A shows a robot 1200 having a videoprojector 1201 with a projection axis 1202 forming an angle Θ of 90°with wall 1204. A projection axis is an imaginary line that describes anaverage directionality of light rays associated with a projection, or acenter of such projection, or a line that generally describes adirectionality of the projection. As a result, the video projector 1201is square with the wall 1204 and projected images are not distorted. Incontrast, robot 1200 in FIG. 12B is shown to be oriented such that theprojection axis 1202′ form an angle Θ′ with the wall 1204 that is not90° (e.g., perpendicular). As a result, images that are projected,without correction, may be skewed or distorted, because the projectionaxis 1202′ is not perpendicular with the wall 1204.

FIG. 12C illustrates one way robot 1200 is able to orient videoprojector 1201 to be square with wall 1204. Robot 1200 is shown to senda pulse 1206 to measure a distance from wall 1204. The robot 1200 movesin various directions until it moves in a direction that a subsequentpulse 1206′ indicates that the robot maintains a similar distance awayfrom the wall 1204 as it travels along that direction. The pulses mayinclude an acoustic pulse, IR light, laser light, or otherelectromagnetic signal for detecting distance. In FIG. 12D, the robot1200 rotates while emitting a continuous test pulse for finding anorientation in which the pulse is reflected back the most. In FIG. 12E,the robot 1200 accesses a map 1210 of the physical environment. Therobot 1200 is able to determine its position 1200′ in map 1210 via GPSand its orientation via compass. As a result, the robot 1200 can orientits video projector to be square with wall 1204 (e.g., by having aprojection axis that is perpendicular). As a result, an additionaltechnological benefit of a self-aligning projector extends from themethods and system described here and embodied by a RUID.

FIG. 13 illustrates an additional embodiment of a RUID 1300 that may beused with the presented method and/or system. Robot 1300 includeshardware such as processor 1302, battery 1304, facial recognition 1306,buttons, sensors, switches 1308, sound localization 1310, display 1312,and memory 1314. Robot 1300 is also shown to include a position module1326 that comprises a magnetometer 1316, an accelerometer 1318, agyroscope 1320, a GPS 1322, and a compass 1324. Further included inrobot 1300 are speakers 1328, microphone 1330, LEDs 1332, object/s forvisual recognition 1334, IR lights 1336, front camera 1338, rear camera1340, gaze tracking camera/s 1342, USB 1344, permanent storage 1346,vibro-tactile feedback 1348, communications link 1350, WiFi 1352,ultra-sonic communication 1354, Bluetooth 1356, near field communication(NFC) 1358, depth sensor 1360, video projector 1362, locomotion unit1364, and proximity sensor 1366, and robotic arm controller 1368.

Although the method operations were described in a specific order, itshould be understood that other housekeeping operations may be performedin between operations, or operations may be adjusted so that they occurat slightly different times, or may be distributed in a system whichallows the occurrence of the processing operations at various intervalsassociated with the processing.

One or more embodiments can also be fabricated as computer readable codeon a computer readable medium. The computer readable medium is any datastorage device that can store data, which can be thereafter be read by acomputer system. Examples of the computer readable medium include harddrives, network attached storage (NAS), read-only memory, random-accessmemory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes and other optical andnon-optical data storage devices. The computer readable medium caninclude computer readable tangible medium distributed over anetwork-coupled computer system so that the computer readable code isstored and executed in a distributed fashion.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, it will be apparent thatcertain changes and modifications can be practiced within the scope ofthe appended claims. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and the embodiments arenot to be limited to the details given herein, but may be modifiedwithin the scope and equivalents of the appended claims.

What is claimed is:
 1. A method for providing an interactive interfaceby a robot, the method comprising: identifying a position of a user in aphysical environment and a point of gaze (POG) of the user; identifyinga surface within the physical environment for projecting the interactiveinterface based on the position of the user, wherein the identifiedsurface is updated based on an update in the position of the user andthe POG of the user; the robot moving to a location within the physicalenvironment based on the position of the user, the POG of the user, andthe surface for projecting the interactive interface; projecting ontothe surface the interactive interface, the interactive interfaceincluding a plurality of input options, responsive to detecting a changein the POG of the user, moving the robot such that the interactiveinterface is continuously projected in the direction of the POG in thephysical environment; capturing a plurality of images of the interactiveinterface while the interactive interface is being interacted with bythe user; and determining, from the plurality of images, an input optionassociated with a selection made by the user, the input option beingcommunicated to a server for processing the selection.
 2. The method ofclaim 1, further comprising: providing, to the interactive interface, afeedback that the input option associated with the selection made by theuser has been made.
 3. The method of claim 1, wherein projecting theuser interface is performed by a projection device having a lens systemfor focusing light to render images associated with the interactiveinterface onto the surface.
 4. The method of claim 1, wherein theprojecting the user interface includes projecting a graphical userinterface (GUI) associated with the interactive interface and graphicalselection buttons associated with the plurality of input options.
 5. Themethod of claim 1, wherein said identifying the surface within thephysical environment includes analyzing attributes of the surface, theattributes including one or more of reflectivity, contrast, color,texture, area, gain, or surface contours.
 6. The method of claim 1,further comprising: correcting for skew in an appearance of theinteractive interface during said projecting the interactive interface,the correcting including distorting one or more images associated withthe user interface before being projected onto the surface.
 7. Themethod of claim 1, further comprising: correcting for a skew inappearance of the interactive interface during said projecting theinteractive interface, the correcting including one or both of moving toa second location or changing an orientation for projecting theinteractive interface, or adjusting the location of the robot, or acombination thereof.
 8. The method of claim 1, further comprising:detecting a diminution of contrast associated with the interactiveinterface based on an analysis of one or more colors associated with thesurface; and correcting for the diminution of contrast by adjusting oneor more colors associated with the interactive interface based on theanalysis of the one or more colors associated with the surface.
 9. Themethod of claim 1, wherein said determining, from the plurality ofimages, the input option associated with the selection made by the userincludes detecting a touch made by the user on the surface in a regionassociated with the input option of the interactive interface.
 10. Themethod of claim 9, wherein said determining the input option associatedwith the selection made by the user further includes capturing a soundassociated with the touch made by the user on the surface.
 11. Themethod of claim 1, wherein said determining the input option associatedwith the selection made by the user includes detecting a hand gesture, ahand position, a hand motion, a finger position, a finger motion, a handplacement, a finger placement, or a hand sign, or a combination of twoor more thereof.
 12. The method of claim 1, further comprising:receiving, from the server in response to said processing the selection,additional content for the projecting onto the surface.
 13. A method forproviding a gaming session by a robot, comprising: receiving, from auser, an instruction for executing a video game; identifying a positionof the user within the physical environment and a point of gaze (POG) ofthe user; identifying attributes of a surface within the physicalenvironment for projecting the video game; determining a game controllerfrom a plurality of game controllers to be used by the user forinteracting with the video game; establishing a wireless connection withthe game controller; the robot moving to a first location proximal tothe user for providing the game controller to the user based on theposition of the user; the robot moving to a second location forprojecting a graphical output of the video game based on the position ofthe user, the POG of the user, and the attributes of the surface; andexecuting the video game and projecting the graphical output of thevideo game onto the surface, wherein the game controller enables theuser to interact with the video game, wherein the surface is updatedbased on an update in the position of the user and the POG, responsiveto detecting a change in the POG of the user, moving the robot such thatthe graphical output of the video game is continuously projected in thedirection of the POG in the physical environment.
 14. The method ofclaim 13, wherein said determining the game controller from a pluralityof game controllers is based on one or more of a request made by theuser, a preference setting of the user, a prediction, a degree ofrandomness, or a specification provided by the video game.
 15. Themethod of claim 13, wherein the game controller is a detachable partthat is mounted on the robot.
 16. The method of claim 13, wherein saidproviding the game controller to the user is performed by a robotic armof the robot.
 17. The method of claim 13, wherein the game controller tobe used by the user for interacting with the video game is a robotic armof the robot.